Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


Tailoring  Multimedia  Instruction  to  Soldier  Needs 


Thomas  Rhett  Graves,  Ph.D. 
U.S.  Army  Research  Institute 
Fort  Benning,  GA 
Thomas.R.Graves5.civ@mail.mil 


Paul  N.  Blankenbeckler,  Richard  L.  Wampler 
Northrop  Grumman  Technical  Services 
Columbus,  GA 

Paul.Blankenbeckler@ngc.com, 

Rich.Wampler@ngc.com 


ABSTRACT 

To  address  particular  learning  needs,  instruction  should  be  designed  to  provide  the  right  information  to  the  right  learner 
at  the  right  time.  While  interactive  multimedia  instruction  can  reach  a  large  audience,  different  learners  have  different 
learning  needs.  In  this  research,  three  questions  were  addressed:  (a)  how  could  existing  Army  interactive  multimedia 
instruction  (IMI)  be  modified  to  a  needs-based  format,  (b)  what  types  of  instructional  design  techniques  could  be 
applied  to  design  IMI  for  learners’  specific  needs,  and  (c)  what  outcomes  are  associated  with  different  types  of  needs- 
focused  IMI?  Following  a  survey  of  existing  Combat  Arms  IMI,  it  appeared  that  most  would  require  modification  for 
reuse  in  a  needs-based  context.  Six  IMI  exemplars  were  developed  for  two  topics  (i.e.,  Adjust  Indirect  Fire  and 
Conduct  a  Defense  by  a  Squad)  targeting  the  learning  needs  of  new  squad  or  team  leaders.  The  IMI  were  developed 
for  three  needs-focused  conditions  to  test  whether  learners  with  differing  levels  of  prior  knowledge  and  experience 
performed  better  with  different  types  of  IMI.  One  condition  was  tailored  training,  and  the  other  two  were  not  tailored, 
and  were  designated  as  familiarization  and  core/refresher  training.  The  exemplars  were  tested  with  Soldiers  attending 
the  Warrior  Leader  Course  at  Fort  Benning.  In  all  conditions,  Soldiers  showed  higher  test  scores  after  training. 
However,  for  the  less  familiar  topic,  the  greatest  impact  was  found  for  the  tailored  training  condition  over  non-tailored 
familiarization  and  core/refresher  training.  The  familiar  topic  showed  no  differences  among  needs-based  conditions. 
These  results  indicated  that  structure  is  necessary  for  novel  material  to  mimic  what  learners  naturally  do  with  familiar 
material.  To  effectively  modify  IMI  to  a  needs-based  format,  one  needs  to  define  the  learning  needs  of  the  specific 
audience  up  front  and  to  structure  the  IMI  to  support  individual  choice  and  flexibility. 
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INTRODUCTION 

Dyer,  Singh,  and  Clark  (2005)  found  that  Soldiers  who  were  less  experienced  in  a  topic  domain  required  training  that 
was  more  structured  than  Soldiers  who  were  more  experienced,  with  the  latter  group  benefiting  from  being  able  to 
exercise  greater  control  over  how  and  what  they  were  learning.  Soldiers  in  training  embody  a  variety  of  educational 
backgrounds,  professional  experiences,  and  learning  needs.  From  the  perspective  of  the  Army  Learning  Model 
(ALM),  interactive  multimedia  instruction  (IMI)  is  viewed  as  an  essential  tool  to  deliver  instruction  when,  where,  and 
how  it  is  needed  to  support  learner-centered  and  life-long  learning  initiatives  (TRADOC,  2011).  While  IMI  is  able  to 
reach  a  large  audience,  it  may  not  always  deliver  instruction  appropriate  to  the  diverse  knowledge  and  experience 
represented  within  that  audience.  The  when  and  the  where  are  questions  of  accessibility,  whereas  the  how  is  a 
question  that  focuses  on  the  needs  of  particular  learners.  Adaptive  IMI  has  been  developed  to  address  the  how 
question,  but  many  existing  examples  were  expensive  and  complex  to  develop,  emphasizing  high-bandwidth  features. 
IMI  must  be  designed  to  meet  the  needs  of  learners  with  varying  levels  of  knowledge  and  experience,  who  may  be 
seeking  training  in  response  to  their  unique  job  demands.  Quite  often  it  must  do  so  in  a  way  that  can  be  easily  applied 
by  Army  training  developers  and  distributed  to  the  Army.  To  address  this  problem,  our  research  focused  on  (a) 
surveying  existing  Army  Combat  Arms  IMI  to  determine  whether  and  how  it  could  be  modified  to  needs-based 
formats,  (b)  identifying  and  applying  instructional  design  techniques  to  develop  IMI  exemplars  addressing  specific 
learning  needs,  and  (c)  determining  whether  different  types  of  needs-based  IMI  are  associated  with  measurable 
learning  outcomes  among  students  with  varying  familiarity  with  the  training  domain. 

Three  types  of  needs-focused  IMI  designs  were  addressed,  including  two  non-tailored  variations  (i.e.,  familiarization 
and  core/refresher)  and  a  tailored  training  variation.  Familiarization  focused  broadly  on  a  topic  domain,  enabling 
learners  to  form  a  coherent  understanding  of  the  domain  but  not  going  into  great  depth  on  any  single  topic. 
Core/refresher  training  focused  specifically  on  a  task  and  provided  in-depth  information  on  how  to  execute  the  task; 
there  was  much  less  breadth  (or  contextual)  information  provided.  Finally,  tailored  training  combined  both  breadth 
and  depth  of  information  into  multiple,  learner-selected  learning  paths.  In  the  tailored  condition,  a  diagnostic 
assessment  with  individualized  feedback  was  also  provided  to  learners  before  and  after  training  to  help  inform  their 
decision  to  pursue  a  particular  learning  path  among  those  available.  We  hypothesized  that  learners  in  the  tailored 
training  IMI  condition  would  benefit  more  than  those  in  the  non-tailored  familiarization  and  core/refresher  conditions, 
and  that  tailored  training  would  most  benefit  learners  in  the  less  familiar  than  in  the  more  familiar  topic  domain. 

Overview  of  the  Research  Process 

This  research  was  conducted  in  multiple  phases.  In  the  first  phase,  we  reviewed  the  literature  and  collected  examples 
of  existing  Army  IMI  to  evaluate  whether  and  how  existing  Army  IMI  was  addressing  particular  learning  needs. 
Moreover,  the  central  concern  was  whether  existing  IMI  could  be  modified  to  be  needs-focused  format,  i.e.,  targeting 
a  particular  audience  with  job-specific  and  individual  learning  requirements.  The  second  phase  was  focused  on 
developing  IMI  exemplars  that  would  reflect  an  instructional  design  and  content  selection  that  was  focused  on  the 
particular  needs  of  squad/team  leaders.  The  topic  domain  selected  was  Combat  Arms  (e.g.,  Infantry,  Air  Defense, 
Armor,  Field  Artillery)  and  the  particular  topics  wer z  Adjust  Indirect  Fire  and  Conduct  a  Defense  by  a  Squad.  Finally, 
the  needs-focused  IMI  exemplars  were  developed  and  tested  with  Soldiers  enrolled  in  the  Warrior  Leader  Course 
(WLC)  at  Flenry  Caro  Noncommissioned  Officer  Academy  (NCOA)  at  Fort  Benning,  GA.  Figure  1  provides  an 
overview  of  the  research  process. 
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Figure  1.  Overview  of  the  Research  Process. 


INSTRUCTIONAL  DESIGN 

Published  literature  was  reviewed  to  identify  general  features  of  good  training  design.  Much  of  the  empirical  research 
on  IMI  design  consists  of  studies  that  focused  on  manipulating  specific  design  features  (e.g.,  highlighting  text,  or 
varying  communication  style)  and  evaluating  their  effects  on  learning  outcomes.  Based  on  this  work,  researchers  have 
prescribed  guidelines  to  use  in  the  design  of  IMI  (see  Clark  &  Mayer,  2008).  Mayer  and  his  colleagues’  indicated  an 
effective  instructional  design  reduces  irrelevant  information  and  helps  learners  to  think  more  deeply  about  the  training 
domain  by  addressing  three  design  goals:  (a)  reducing  extraneous  cognitive  processing,  (b)  managing  essential 
cognitive  processing,  and  (c)  facilitating  generative  processing  (Mayer,  2005).  The  features  that  are  focused  on 
extraneous  cognitive  processing  are  aimed  to  reduce  distractions  and  make  critical  information  stand  out  to  learners. 
These  design  techniques  are  focused  on  selecting  the  right  words  and  images,  making  sure  critical  ideas  stand  out,  not 
overloading  the  learner,  and  following  a  logical  use  of  visual  space  and  training  time.  Applying  these  techniques 
likely  helps  learners  better  identify  and  learn  relevant  content.  Features  that  help  to  manage  essential  cognitive 
processing  focus  on  helping  the  learner  to  make  sense  of  the  information  they  are  learning.  This  goal  focuses  on 
related  principles  that  allow  learners  to  pace  their  learning,  make  use  of  preexisting  knowledge,  and  reduce  attention 
conflicts  within  single  sensory  modalities  (e.g.,  not  presenting  an  animation  with  static  text  because  both  are  within 
the  visual  modality).  The  final  set  of  features  relate  to  facilitating  generative  processing,  and  are  intended  to  support 
learners  in  achieving  a  deeper  understanding  of  the  material  they  are  learning.  This  third  goal  is  to  reduce  as  much  as 
possible  barriers  to  understanding  by  establishing  an  appropriate  communication  style  with  a  learner  (see  Clark  & 
Mayer,  2008;  Mayer,  2009;  Moreno  &  Mayer,  2007).  These  goals  and  techniques  informed  the  review  of  existing 
IMI  in  the  first  phase  of  this  research  and  when  designing  and  developing  IMI  exemplars  in  the  second  phase. 

PHASE  I:  REVIEW  OF  EXISTING  ARMY  IMI 

In  the  first  phase  of  this  research,  179  examples  of  existing  Combat  Arms  IMI  were  collected  and  reviewed.  The 
examples  were  gathered  from  the  Directorates  of  Training  and  Doctrine  at  the  Maneuver  Center  of  Excellence  (Fort 
Benning,  GA),  the  Fires  Center  of  Excellence  (Fort  Sill,  OK),  and  the  Iowa  National  Guard  Distributed  Learning 
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Development  Center  (Camp  Dodge,  IA).  Also  reviewed  were  examples  of  IMI  from  the  Army  Knowledge  Online 
My  Training  Tab  (MT2)  site. 

The  examples  were  initially  evaluated  for  how  relevant  they  would  be  for  a  new  squad  or  team  leader.  Once  the 
candidates  for  repurposing  were  identified,  four  key  criteria  guided  our  decisions:  (a)  whether  source  files  were 
available,  including  uncompiled  and  editable  files,  (b)  whether  source  materials  were  compatible  with  the  software 
available  to  modify  files,  (c)  whether  the  concepts,  uniforms,  equipment,  and  materials  depicted  were  current,  and  (d) 
the  instructional  design  characteristics  of  the  existing  IMI  (e.g.,  content  selection;  quality  of  graphics,  animation,  text, 
and  narration;  navigation/organization).  Initially,  the  plan  was  to  redesign  existing  IMI  to  produce  exemplars  of 
needs-focused  IMI.  Ultimately,  however,  only  30%  was  amenable  to  being  repurposed  (see  Blankenbeckler,  Graves, 
&  Wampler,  2013).  Most  of  the  existing  IMI  lacked  source  files  that  could  be  modified,  or  had  been  designed  in 
superseded  versions  of  software.  In  addition,  much  of  the  existing  IMI  was  designed  to  focus  on  a  general  audience, 
lacking  the  details  necessary  to  focus  content  on  squad  or  team  leaders’  specific  learning  needs.  While  other 
organizations  have  reported  similar  difficulties  reusing  IMI,  Shanley  et  al.  (2009)  have  found  that  25%  of 
organizations  who  participated  in  their  research  reported  a  positive  return  on  investment  in  terms  of  time,  labor,  or 
dollars  from  reusing  IMI.  However,  the  majority  of  organizations  reported  lower  than  expected  returns,  and  35% 
reported  no  savings  or  a  loss. 

PHASE  II:  DESIGNING  AND  DEVELOPING  NEEDS-FOCUSED  IMI 

After  reviewing  existing  IMI,  two  topics  were  identified  around  which  to  design  six  IMI  exemplars.  Given  the 
emphasis  of  the  MCOE  initiative  Squad  as  the  Foundation  of  the  Decisive  Force ,  two  topic  domains  were  selected 
that  would  be  relevant  to  new  squad  and  team  leaders  in  combat  arms  specialties  (Maneuver  Center  of  Excellence, 
N.D.).  These  two  domains  were:  Adjust  Indirect  Fire  (Task  #061-283-6003)  and  Conduct  a  Defense  by  a  Squad  (Task 
#071-430-0002)  (U.S.  Army,  2008).  Table  1  describes  the  six  IMI  exemplars  in  relation  to  the  selected  topic  domains. 


Table  1.  Six  Point  of  Need  IMI  Exemplars  by  Topic,  Type  of  IMI,  and  Title 


Topic 

Type  of  IMI 

Title 

Adjust  Indirect  Fire 
(Task  #061-283-6003) 

Familiarization 

Engaging  Targets  with  Supporting  Fires 

Core 

Conduct  Immediate  Suppression 

Tailored  Training 

Adjust  Indirect  Fire 

Conduct  a  Defense  by  a 
Squad 

(Task  #071-430-0002) 

Familiarization 

Prepare  Positions  for  Crew-Served  Weapons  During  an 
Urban  Operation 

Core 

Designate  and  Prepare  Urban  Fighting  Positions  for  a 
Javelin  Team 

Tailored  Training 

Conduct  a  Defense  by  a  Squad  in  an  Urban  Operation 

Note:  The  tasks,  conditions,  and  standards  for  the  selected  topics  are  documented  in  the  Soldier’s  Manual  of  Common 
Tasks,  Warrior  Leader  Skills  Level  2,  3,  and  4  (U.S.  Army,  2008). 


Given  that  the  target  audience  was  new  squad  or  team  leaders  (i.e.,  Specialist/Corporals,  E-4,  and  Sergeants,  E-5),  it 
was  anticipated  that  Conduct  a  Defense  by  a  Squad  would  be  more  familiar  to  Soldiers,  and  Adjust  Indirect  Fire  would 
be  less  familiar.  Conduct  a  Defense  by  a  Squad  is  a  common  task  in  which  Soldiers  at  varying  levels  participate,  even 
if  they  are  not  in  the  role  of  decision-makers  or  leaders.  For  similar  reasons,  it  was  anticipated  that  Adjust  Indirect 
Fire  would  be  less  familiar  given  its  complexity  and  more  specialized  training  requirements. 
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The  three  types  of  training — familiarization,  core/refresher,  and  tailored — were  selected  to  cover  a  variety  of  training 
needs  from  preparation  for  a  course,  acquiring  background  knowledge  to  support  more  in-depth  study,  supporting 
execution  of  tasks  on  the  job,  etc.  Familiarization  and  core/refresher  training  were  not  tailored,  whereas  the  tailored 
training  condition  was  designed  to  facilitate  learners  in  selecting  an  individualized  learning  path.  Familiarization 
training  was  intended  to  provide  learners  with  an  overview  of  the  topic  and  resources  to  learn  more,  i.e.,  breadth  of 
information.  It  would  assist  learners  in  gaining  a  modest  understanding  of  the  topic  domain,  but  not  a  high-level  of 
proficiency  or  expertise.  Familiarization  1MI  was  further  intended  for  learners  who  need  to  be  prepared  for  subsequent 
and  more  in  depth  learning,  assisting  them  in  forming  a  basic  understanding  of  the  content  domain. 

Core/refresher  training  was  intended  to  provide  only  the  essential  information  learners  would  need  to  perform  a 
defined  task,  i.e.,  it  provided  depth  of  information,  with  less  breadth.  When  designing  core/refresher  IMI,  the  model 
was  ‘how-to’  manuals  that  present  step-by-step  how  to  execute  a  task.  Again,  given  that  the  focus  was  on  learners’ 
immediate  needs,  they  were  not  intended  to  reach  a  high-level  of  proficiency.  Core/refresher  IMI  was  designed  for 
learners  who  would  need  immediate  support  in  conducting  on-the-job  tasks  or  a  quick  refresher  of  a  previously  learned 
skill.  This  type  of  training  would  not  lead  to  mastery  of  the  task,  as  the  core/refresher  IMI  contains  no  hands-on 
performance  or  sophisticated  simulation. 

Finally,  tailored  training  combined  both  breadth  and  depth  of  information  into  multiple  user-selected  learning  paths. 
It  also  included  pre-  and  post-training  diagnostic  assessments  with  individualized  feedback  following  each  assessment. 
Feedback  to  learners  was  based  on  a  pre-training  diagnostic  assessment.  The  feedback  was  intended  to  help  learners 
identify  content  areas  in  which  they  were  deficient  and  to  select  among  the  available  learning  paths.  After  learners 
completed  training,  they  were  again  able  to  take  a  diagnostic  assessment  and  receive  feedback,  with  the  option  for 
additional  training.  The  tailored  training  design  was  intended  to  meet  a  variety  of  learning  needs.  The  diagnostic 
assessments  were  intended  to  help  learners  to  be  more  aware  of  their  specific  learning  needs  and  to  make  reasonable 
decisions  on  how  to  tailor  their  learning  experience.  While  this  approach  to  tailoring  did  not  make  use  of  cutting-edge 
technologies  (such  as  computer-adaptive  testing),  it  represented  an  effective  and  low-cost  solution  to  enable  tailored 
training  in  Army  IMI. 

In  designing  the  IMI  exemplars  for  the  experiment,  configurations  of  design  principles  were  applied.  One  design 
feature  shared  across  the  different  types  of  IMI  was  content  chunking  in  terms  of  tasks  and  parts  of  tasks  (see  Figure 
2). 
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Figure  2.  Examples  of  Content  Chunking  from  Each  Domain 

Content  chunking  allowed  us  to  present  an  organized  structure  to  the  learner  up  front  as  well  as  consolidate  closely 
related  material  into  manageable  parts,  helping  to  manage  cognitive  load  (Sweller,  2005)  and  reduce  extraneous 
processing  (Mayer,  2005).  This  structure  was  intended  to  facilitate  learners  in  navigating  through  the  IMI,  serving  as 
way  to  organize  content  for  the  less  knowledgeable  Soldiers  or  as  a  road-map  for  the  more  knowledgeable.  The 
assumption  was  that  Soldiers  could  use  their  prior  knowledge  and  experience  to  identify  parts  of  the  IMI  that  were 
particularly  relevant  to  them. 


2014  Paper  No.  14049  Page  5  of  12 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


Making  sure  the  instruction  had  a  narrative  pattern,  presenting  a  sequence  of  steps  to  interconnect  parts  of  the  tasks, 
was  also  important  (Clark,  2005).  This  approach  was  particularly  useful  for  the  core/refresher  variations  of  needs- 
focused  1MI.  For  example,  questions  and  feedback  followed  the  sequence  of  an  adjust  fire  mission  to  summarize  the 
call  for  fire  and  Fire  Direction  Center  (FDC)  responses  (see  Figure  3). 
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Figure  3.  Example  of  Narrative  Flow 


Evaluation  and  feedback  on  specific  task  steps  was  also  expected  to  help  guide  learners  through  the  learning  process, 
and  served  as  a  particular  way  of  implementing  narrative  sequencing  (see  Figure  4).  This  approach  helped  learners  to 
test  their  knowledge  throughout  the  training  and  to  maintain  their  focus  on  relevant  content.  This  was  viewed  as  an 
alternative  way  to  approach  to  checks-on-learning  that  was  intended  to  maintain  focus  on  task  execution.  The  learner 
was  questioned  and  responded  for  each  step  in  the  task  as  it  builds  in  complexity.  Subsequently,  they  are  presented 
feedback  on  their  decisions  between  steps,  building  toward  the  complete  task  (van  Merrienboer  &  Kester,  2005). 


Figure  4.  Example  of  Questions  and  Feedback  on  Individual  Task  Steps 


Finally,  the  tailored  training  variation  of  needs-based  IMI  has  some  of  its  own  unique  features.  For  instance,  whole 
task  evaluation  and  feedback  was  unique  to  this  variation.  The  tailored  training  IMI  was  designed  with  a  diagnostic 
assessment  at  the  start.  This  diagnostic  was  designed  to  measure  prior  knowledge  learners  had  concerning  the  content 
domain,  and  then  to  provide  feedback  to  learners  on  their  deficiencies.  It  was  structured  such  that  learners  would 
receive  individualized  feedback  on  particular  content  areas  that  they  should  focus  on  in  training.  Learners  were 
allowed  to  select  the  content  they  wanted  to  look  at,  although  they  were  given  recommendations  to  optimize  their  self- 
selected  learning  path.  Training  recommendations  were  made  in  relation  to  the  structure  of  chunked  content  within 
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the  IMI,  enabling  learners  to  identify  how  to  navigate  through  the  content.  This  simple  approach  enabled  us  to  adapt 
training  to  learners’  needs  without  employing  a  complex  learning  management  system,  as  the  learner  ultimately 
determined  how  they  would  proceed  through  the  IMI.  Once  the  IMI  was  complete,  learners  were  again  given  an 
opportunity  to  test  their  knowledge  and  were  provided  with  additional  feedback  to  help  guide  their  selection  of  future 
learning  (see  Figure  5). 


1.  Dcsignalo/propare/adjust 

2.  Designate/prepare: 

primary  individual/taam/ 

•  Alternato  positions 
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•  Supplementary 

-iJ 

positions 

id 

a 

(These  positions  should 

Provide  effective/ 

meet  (lie  same  standards 

adequate  fields  of  fire. 

a 

as  primary  positions.) 

3.  Employ  fire  control 

a 

a 

measures  effectively 

Enhance  Sddier/unit 

a 

a 

survival  and  safety 

a 

a 

a 
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Figure  5.  Example  of  Whole-Task  Diagnostic  Assessment  and  Feedback  in  Tailored  Training  Condition 
PHASE  III:  EXPERIMENTAL  EVALUATION  OF  THE  IMI 
Method 

In  the  final  phase  of  this  research,  the  IMI  were  tested  with  N  =  91  Soldiers  enrolled  in  the  WLC  at  the  Henry  Caro 
Noncommissioned  Officer  Academy  at  Fort  Benning,  GA.  The  experimental  design  was  organized  around  two  topics 
(i.e.,  more  familiar  vs.  less  familiar)  and  three  types  of  needs-based  IMI.  These  types  were  non-tailored  (i.e., 
familiarization  and  core/refresher)  and  tailored  training.  The  IMI  were  tested  across  six  two-hour  sessions,  with  all 
three  IMI  variations  tested  with  different  participants  in  each  session.  There  were  15  to  19  WLC  students  who 
participated  in  each  session.  At  the  start  of  each  session.  Soldiers  were  briefed  on  the  purpose  of  the  research  and 
administered  informed  consent.  They  were  then  asked  to  complete  a  short  demographic  questionnaire  and  to  rate  their 
familiarity  with  the  training  topic  for  that  session,  described  broadly  as  either  Adjust  Indirect  Fire  or  Conduct  a 
Defense  by  a  Squad.  Based  on  their  ratings,  Soldiers  were  assigned  to  one  of  the  three  IMI  conditions:  familiarization 
(32%),  core/refresher  (32%),  or  tailored  training  (36%).  Self-rated  background  knowledge  was  used  to  balance 
assignment  of  Soldiers  to  each  group.  Soldiers  who  participated  were  predominately  Specialists  (E-4;  67%),  and  most 
had  deployment  experience  (75%).  Some  had  experience  with  the  relevant  domains,  with  fewer  having  experience 
with  Adjust  Indirect  Fire  (16%)  than  with  Conduct  a  Defense  by  a  Squad  (38%).  Both  Combat  Arms  (31%)  and  other 
MOSs  (69%)  were  represented. 

Once  assigned  to  their  training  condition,  each  Soldier  was  administered  a  pretraining  test  for  either  Adjust  Indirect 
Fire  or  Conduct  a  Defense  by  a  Squad.  Each  topic  has  two  test  forms  that  were  matched  for  content  and  difficulty. 
These  forms  were  alternated  between  pre-  and  post-training  (i.e.,  if  a  Soldier  received  Form  A  before  training,  he  or 
she  received  Form  B  after  training).  Once  their  test  was  complete,  it  was  turned  in  to  a  data  collector,  the  time  was 
recorded,  and  they  were  allowed  to  begin  training  with  the  IMI  at  their  own  pace.  At  the  completion  of  training. 
Soldiers  once  again  completed  the  alternate  form  of  a  knowledge  assessment.  Finally,  Soldiers  were  asked  to  fill  out 
a  user  evaluation  of  the  IMI;  a  few  of  the  Soldiers  were  briefly  interviewed  following  the  session. 

Results 

We  hypothesized  that  all  IMI  conditions  would  improve  Soldiers’  performance,  with  tailored  training  showing  the 
most  benefit  regardless  of  topic  domain;  additionally,  we  hypothesized  that  tailored  training  would  most  benefit 
Soldiers  who  were  in  the  less  familiar  topic  condition.  To  verify  our  assumption  that  Soldiers  would  have  different 
background  knowledge  and  experience  with  the  respective  topic  domains,  we  looked  at  how  Soldiers  rated  their  prior 
knowledge  (see  Table  2).  Conduct  a  Defense  by  a  Squad  ( M  =  5.04;  sd  =  2.19)  rated  higher  than  Adjust  Indirect  Fire 
(M  =  3.30;  sd  =  2.15),  F(  1,  89)  =  14.73,  p  <  0.001.  This  finding  supported  our  initial  assumption.  Moreover,  more 
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Soldiers  indicated  having  prior  experience  with  the  Conduct  a  Defense  by  a  Squad  domain  (38%)  compared  with 
Adjust  Indirect  Fire  (\6%),X2(N  =  91)  =  5.72, p  =  0.017.  The  IMI  were  designed  to  take  roughly  an  equivalent  time 
to  complete,  with  an  exception  of  the  diagnostic  testing  in  the  tailored  training  condition.  The  amount  of  time  that 
Soldiers  spent  on  the  training  was  roughly  equivalent  (M=  12  to  16  minutes;  sd  =  7  to  8  minutes)  across  conditions 
after  subtracting  the  additional  time  for  the  diagnostic  assessment  in  the  tailored  training  condition  (M  =  15  to  18 
minutes;  sd=  3  to  5  minutes).  The  exception  to  this  finding  was  for  core/refresher  training  ( M  =  23  minutes;  sd  =  7 
minutes)  in  the  Conduct  a  Defense  by  a  Squad  condition,  Pst<mdardized=  0.35,  t  =  2.49,  p  =  0.016,  partial  r  =  0.35.  This 
core/refresher  IMI  was  focused  on  preparing  urban  fighting  positions  for  the  Javelin  missile  team. 


Table  2.  Soldiers’  Mean  Scores  by  Training  Topic  and  Type  of  Needs-Focused  Training 


Measures 

Adjust  Indirect  Fire 
(Mean) 

Conduct  a  Defense 
(Mean) 

Fam 

Core 

Tailored 

Fam 

Core 

Tailored 

Rating  of  Prior  Knowledge* 

2.9 

3.4 

3.5 

5.3 

5.3 

4.6 

Training  Time  (Mean  Minutes); 

12m 

16m 

16m 

16m 

23m 

16m 

Embedded  Assessment  Time 

— 

— 

18m 

— 

— 

15m 

Pretest  Score  (Mean  %  Items  Correct) 

39% 

34% 

48% 

51% 

51% 

45% 

Posttest  (Mean  %  Items  Correct) 

47% 

44% 

66% 

60% 

63% 

56% 

*Note:  The  self-ratings  of  prior  knowledge  were  scaled  as  ‘1  to  3’  =  ‘None  to  Little’,  ‘4  to  6’  =  ‘Moderate’,  and  ‘7 
to  9’  =  ‘Extensive’. 


Stepwise  regression  was  used  to  examine  the  relationship  between  Soldiers’  posttest  scores  (criterion)  and  their  pretest 
scores,  training  time,  type  of  needs-focused  training  they  received,  and  their  self-rating  of  prior  knowledge  for  the 
more  familiar  (Conduct  a  Defense  by  a  Squad)  and  less  familiar  (Adjust  Indirect  Fire)  training  topics.  The  overall 
models  were  significant  for  both  Adjust  Indirect  Fire,  F( 2,  41)  =  33.9 ,p  <  0.001,  and  Conduct  a  Defense  by  a  Squad, 
F( 1,  45)  =  10.17,  p  =  0.003.  (The  degrees  of  freedom  for  Conduct  a  Defense  by  a  Squad  were  reduced  because 
variables  coding  for  differences  among  IMI  types  had  no  effect.)  Pretest  scores  predicted  posttest  scores  for  both 
Adjust  Indirect  Fire  (Ji standardized  .66,  t  =  6.36, p  <0.001, partial  r=  0.61)  and  Conduct  a  Defense  by  a  Squad  (J4 standardized 
=  0.43,  t  =  3.18,  p  <  0.003 ,  partial  r=  0.43).  All  Soldiers  improved  their  performance  at  posttest  regardless  of  whether 
the  training  was  on  the  more  familiar  or  less  familiar  topic  domain.  A  mean  12%  point  increase  was  found  from 
pretest  to  posttest  for  Adjust  Indirect  Fire,  and  a  mean  10.7%  point  increase  for  Conduct  a  Defense  by  a  Squad. 

A  difference  was  found  between  types  of  needs-focused  IMI  for  Adjust  Indirect  Fire  (less  familiar)  topic  domain.  For 
the  less  familiar  domain,  tailored  training  was  associated  with  higher  posttest  scores  (i.e.,  8  to  10  percentage  points), 
over  both  familiarization  and  core/refresher  train  i  ng,  ft  standardized  =  0.25,  t  =  236,p  =  0. 023,  partial  r  =  0.23.  However, 
there  was  no  effect  for  training  type  for  Conduct  a  Defense  by  a  Squad,  nor  were  self-ratings  of  prior  knowledge 
associated  with  posttest  scores  for  either  content  domain.  Figure  6  summarizes  the  results. 

DISCUSSION 

This  research  surveyed  existing  Army  IMI  and  selected  examples  to  modify  for  three  needs-based  training  formats, 
which  were  non-tailored — i.e.,  familiarization  and  core/refresher — and  tailored  training.  Six  IMI  exemplars  were 
developed  and  tested  for  two  topic  domains  that  were  more  and  less  familiar  to  new  squad  or  team  leaders.  In  past 
research.  Dyer,  Singh,  and  Clark  (2005)  found  that  Soldiers  with  less  prior  knowledge  tended  to  benefit  from  having 
more  structured  training,  and  those  who  had  more  prior  knowledge  benefitted  from  being  able  to  adapt  training  to  their 
own  needs.  We  designed  our  IMI  to  be  flexible  in  providing  both  structure  to  guide  learners  as  well  as  a  means  to  be 
tailored  to  Soldiers’  learning  needs.  The  tailored  training  variation  was  the  most  comprehensive  in  terms  of 
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Adjust  Indirect  Fire: 
Tailored  point  gain  (18  pts) 
differed  significantly  from 
core  (10  pts)  and 
familiarization  (8  pts). 


Conduct  a  Defense 
bya  Squad: 

N  o  significant 
difference  in  gains 
between  IMI 
variations. 

Overall  point  gain 
was  statistically 
significant. 


■  Pretest 

■  Posttest 

Note: These  were 
paper-base  dte  sts. 


Fam  Core  Tailored 

Adjust  Indirect  Fire 


Fam  Core  Tailored 

Conduct  a  Defense 


Figure  6.  Pre  and  Post  Test  Scores  by  Content  Domain  and  Training  Type 

the  content  and  structure  provided  to  support  learners  with  less  prior  knowledge.  We  had  hypothesized  that  Soldiers 
in  both  the  more  (Conduct  a  Defense  by  a  Squad)  and  less  (Adjust  Indirect  Fire)  familiar  topic  conditions  would 
benefit  more  from  tailored  training  compared  to  those  in  the  non-tailored  familiarization  and  core/refresher  IMI 
conditions.  We  also  hypothesized  that  Soldiers  in  the  less  familiar  topic  condition  would  benefit  more  from  tailored 
training  than  Soldiers  in  the  more  familiar  topic  condition. 


Our  learner-centered  approach  to  tailoring  was  most  beneficial  to  learners  in  the  less  familiar  content  domain,  but  it 
did  not  appear  to  impact  learners  negatively  in  the  more  familiar  content  domain.  This  finding  was  in  accord  with 
Dyer,  Singh  and  Clark’s  (2005)  earlier  research.  Among  the  many  potential  options  available  for  developing  tailored 
or  adaptive  training,  the  approach  used  here  was  intended  to  be  a  simple  and  straightforward  way  to  incorporate 
tailoring  techniques  into  needs-focused  IMI. 


In  developing  the  IMI  for  this  research,  only  30%  of  the  existing  Army  IMI  review  was  reusable  in  the  way  intended. 
This  finding  suggests  viable  future  research  as  Department  of  Defense  has  emphasized  the  need  to  reuse  IMI  across  a 
variety  of  learning  contexts  (i.e.,  Sharable  Content  Object  Reference  Model;  DoD,  2011).  There  are  many  excellent 
examples  of  Army  IMI  that  demonstrate  solid  instructional  design  and  application  of  learning  science.  However,  few 
are  currently  configured  for  and  are  available  in  a  format  that  facilitates  modification  and  reuse  to  produce  learning 
materials  for  different  courses  and  audiences.  When  seeking  to  modify  and  reuse  existing  IMI,  the  researchers 
encountered  a  number  of  pitfalls.  Often,  source  files  were  difficult  to  track  down,  the  software  in  which  the  original 
IMI  was  designed  was  outdated,  and  uniforms,  doctrine,  etc.,  had  sometimes  changed.  In  order  to  reuse  IMI, 
workarounds  were  often  necessary.  As  an  example,  digitally  recording  audio  and  video  off  the  My  Training  Tab 
(MT2)  site  allowed  us  to  produce  an  editable  file  rather  than  attempting  to  modify  incomplete  or  outdated  source  files. 
Unfortunately,  trying  to  modify  and  reuse  IMI  in  this  way  was  more  time  consuming  than  if  IMI  had  been  developed 
from  a  cold  start. 


When  developing  IMI  in  the  second  phase  of  this  research,  configurations  of  design  features  were  particularly  useful 
for  adapting  content  to  individual  learners’  needs.  For  example,  a  hierarchical  and  chunked  organizational  structure 
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enabled  users  to  quickly  address  a  single  topic  in  relation  to  other  topics  covered  and  to  navigate  through  the  IMI. 
Segmenting  and  presenting  content  in  this  way  may  also  allow  learners  to  take  breaks  and  resume  learning  later  with 
reduced  disruption  (Mayer,  2005).  Many  participants  reported  that  being  able  to  take  a  break  from  training  and  later 
resume  it  would  be  beneficial  to  them,  given  their  many  competing  demands  on  the  job. 

Another  feature  of  the  IMI  was  the  use  of  evaluation  and  feedback  for  each  task  step  when  teaching  a  task.  This  was 
similar  to  the  usual  checks-on-learning  approach  used  by  the  Army,  but  it  focused  on  the  set  of  steps  to  execute  a 
complete  task.  It  was  intended  to  support  learners  in  developing  a  sequential  understanding  of  the  task,  providing 
rationale  for  each  step  and  how  steps  interrelate.  In  order  to  do  this,  the  structure  of  task  steps  and  decision  process 
at  each  point  was  the  central  focus.  This  was  a  particularly  straightforward  approach  for  sequential  tasks  like  Adjust 
Indirect  Fire;  however,  it  was  more  difficult  when  considering  the  multiple  simultaneous  subtasks  and  decision  points 
in  Conduct  a  Defense  by  a  Squad.  The  Conduct  a  Defense  by  a  Squad  IMI  modules  required  more  articulated 
structures  in  order  to  capture  the  various  narrative  threads  of  the  task  and  to  address  learners’  particular  needs.  So, 
although  the  task  was  more  familiar  to  Soldiers,  it  also  required  the  IMI  developed  to  be  more  complex  in  its  design 
and  selection  of  content. 

Finally,  a  diagnostic  assessment  and  individualized  feedback  in  the  tailored  training  condition  was  used  to  assist 
learners  in  identifying  a  path  through  the  IMI  that  would  be  optimized  for  their  learning  needs.  By  providing  detailed 
feedback  to  learners  about  the  topic  areas  in  which  they  had  difficulty  and  allowing  them  to  make  self-aware  decisions 
about  how  they  would  navigate  the  IMI,  it  was  possible  to  apply  tailored  training  principles  in  IMI  without  relying  on 
a  learning  management  system.  While  a  learning  management  system  is  often  required  for  certain  types  of  adaptive 
training,  the  approach  used  in  this  research  may  allow  for  tailoring  within  even  less  complex  and  standalone  types  of 
IMI. 

In  the  experimental  test  of  the  needs-focused  IMI  the  results  for  Adjust  Indirect  Fire,  which  was  the  less  familiar  task, 
were  as  expected.  In  the  Adjust  Indirect  Fire  groups,  all  Soldiers  improved  performance,  with  Soldiers  in  the  tailored 
training  group  showing  much  greater  improvement  over  those  in  the  non-tailored  familiarization  and  core/refresher 
groups.  However,  the  results  for  Conduct  a  Defense  by  a  Squad  did  not  follow  this  pattern.  For  Conduct  a  Defense 
by  a  Squad,  all  three  needs-focused  IMI  conditions  were  associated  with  roughly  equivalent  improvement  in  test 
scores.  One  possible  explanation  for  this  difference  between  the  Adjust  Indirect  Fire  and  Conduct  a  Defense  by  a 
Squad  concerns  the  ways  in  which  the  Soldiers  were  navigating  through  the  IMI.  What  seemed  to  be  happening  more 
frequently  in  the  Conduct  a  Defense  by  a  Squad  group  was  that  Soldiers  tailored  their  learning  experience  by  using 
the  navigation  scheme  regardless  of  IMI  condition.  They  seemed  to  be  using  the  content  headings  and  their  own  prior 
knowledge  of  the  domain  to  track  down  novel  information  more  quickly,  navigating  around — or  quickly  through — 
sections  of  content  that  were  already  familiar  to  them.  With  diagnostic  testing  and  feedback,  the  tailored  training  for 
Adjust  Indirect  Fire  may  have  provided  a  similar  support  to  learners  who  were  less  knowledgeable.  This  support 
perhaps  enabled  them  to  better  tailor  their  learning  experience  to  their  identified  needs  without  having  prior  knowledge 
on  which  to  rely.  Another  possible  explanation  concerns  the  relative  complexity  of  the  tasks.  Adjust  Indirect  Fire, 
while  a  less  familiar  task,  was  also  more  sequential  in  structure  than  Conduct  a  Defense  by  a  Squad.  The  particular 
ways  in  which  the  design  features  were  implemented  may  have  better  supported  learning  of  a  more  sequential  and 
clearly  structured  task.  Even  so,  additional  research  may  be  necessary  to  identify  the  precise  mechanisms  associated 
with  this  effect. 

CONCLUSIONS 

Continued  research  and  innovation  is  essential  concerning  how  to  best  develop  IMI  that  can  be  modified  and  reused 
across  different  learning  contexts  and  for  different  learning  needs.  The  design  features  identified  by  this  research 
provide  a  simple  and  effective  means  to  apply  tailored  training  techniques  within  IMI  that  do  not  require  complex 
programming  and  high  bandwidth  features.  This  streamlined  approach  to  needs-focused  IMI  design  may  be  applicable 
for  the  Army  and  other  branches  of  the  military.  When  learners  are  less  familiar  with  a  topic  domain,  it  may  be  useful 
to  provide  them  with  a  clear  navigational  structure,  including  clearly  labeled  chunks  of  content,  to  supplement  their 
understanding  of  the  domain  and  enable  them  to  select  an  appropriate  learning  path — i.e.,  tailoring  their  learning 
experience.  Using  diagnostic  assessments  with  individualized  feedback  may  further  assist  learners  in  discriminating 
between  personally  relevant  and  irrelevant  content  while  learning,  even  if  they  do  not  yet  have  extensive  domain 
knowledge.  When  learners  are  more  familiar  with  a  topic,  the  navigational  structure  appeared  to  assist  them  in  seeking 


2014  Paper  No.  14049  Page  10  of  12 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2014 


novel  information.  In  this  regard,  the  information  seeking  behavior  we  observed  was  similar  to  what  has  been 
observed  in  other  contexts  when  comparing  novices  to  more  knowledgeable  learners  (Kalyuga,  2005). 
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